Methods and apparatus for real-time digitization of three-dimensional scenes

ABSTRACT

This concerns in part the invention disclosed by co-pending application, particularly to robust determination of features in projection patterns. Disclosed are novel methods and apparatus for obtaining range frame coordinates in moving scenes, where optical radiation is projected onto a scene in the form of dots and strips, where reflected radiation is picked up by a camera and range coordinates calculated thereof.

CROSS-REFERENCE TO RELATED APPLICATION

This application is continuation-in-part of application Ser. No.12/748,397 filed on Mar. 27, 2010.

PRIORITY

Priority is claimed to Provisional Application No. 61/425,200, filed onDec. 20, 2010

DESCRIPTION

1. Field of Invention

The present invention relates to general field of three-dimensional(3-D) digitization of physical objects and three-dimensionalenvironments in particular to obtaining 3-D frames of dense measurementsat rates suitable for moving scenes.

Disclosed are improvements and additional embodiments of my “Method andapparatus for high-speed unconstrained three-dimensional digitization”invention disclosed in co-pending application Ser. No. 12/748,397,particularly relating to novel 3-D digitizing techniques.

The invention further concerns digitization of 3-D scenes by opticaltriangulation, where patterns of randomly distributed point-features andcurvilinear strips, are projected onto objects and their identityestablished form a digital image.

2. Background of Invention

Real-time 3-D imaging systems are present in an increasingly diversenumber of applications such as modeling, security, autonomousnavigation, physical measurements and gesture input.

Digitization systems based on structured light and triangulationprinciple measure distance from sensing unit to object surface typicallyacquiring dense measurements representing 3-D coordinates from sensor'sviewpoint.

An important aspect of this invention is the ability to digitize scenes,where relative motion exists (dynamic), with no mechanical coupling orvisual features affixed to scene.

A number of digitizing methods are based on projecting one or more gridpatterns onto objects, taking one or more digital images of illuminatedobject from a plurality of vantage points, and by processing images toobtain surface coordinates.

U.S. Pat. No. 4,802,759 disclose a method where an object is illuminatedby a grid pattern, which requires finding one reference line first,before other lines in projected grid are identified, therefore imposinga substantial smoothness assumption. Method disclosed in my co-pendingapplication has no restriction on object shape, since stripidentification is independent on detecting any other particular strip.

PCT document WO 93/03579 disclose a system comprising two projectors andone camera, where two sets of parallel lines have angularity,periodicity and depth constraints to facilitate identification.According to the method disclosed in co-pending application, both stripsets contribute measurements to frame coordinates, and no restrictionsare imposed on parallelism, periodicity or mutual angularity, comprisinga single radiation projector coupled to one image sensor, where stripsspacing and directions can be dynamically adjusted.

U.S. application No. 2009/0059241 and U.S. Patt. No. 7,646,896 utilizeintensity-coded information, to carry out pattern identification byanalyzing pattern intensity variations, where coded elements have to berecognized unambiguously. The robustness of decoding may be adverselyimpacted by a number of conditions, such as object geometry, texture,local contrast variance, which impose restrictions on suitability.

U.S. Pat. No. 6,751,344 B1 describes a method in which the subject issuccessively illuminated with a matrix of clots from a plurality ofprojectors, where a plurality of images taken from a plurality ofvantage points is analyzed, to extract the contour of an object.

U.S. Application 20100118123 A1, whose disclosure is incorporated hereinby reference, teaches of a speckle projector where a dot pattern isobtained from a diffractive optical element, and where scene depth isobtained from analyzing pattern shifts relative to a reference image.The shifts are indicative of depth changes and calculated by means ofblock correlation in frequency domain.

Because the speckle points appear at fixed spatial locations, patterndistribution cannot be controlled dynamically. More importantly, certainobjects may exhibit features where strips projection may be moresuitable to extract local geometry, as speckle patterns can createfeature round-offs, detail distortions or miss details entirely, whichis unsuitable for certain applications.

The present invention introduces novel sampling methods where projectedpoint-features and strips may be combined according to necessities.

SUMMARY OF THE INVENTION

This document contains further disclosure on my co-pending applicationSer. No. 12/748,397, to address following aspects:

1. Robust disambiguation of crossing (nodal) points when epipolar testreturns multiple matches, which may arise when at least two nodals sharesame epipolar band. An epipolar band is the region along epipolar linesfor which point coordinates inside the regions are considered to lay onrespective epipolar. Epipolar “collision” is caused in general by imagenoise, projector-camera orientation, or particular strips configuration.

A novel arrangement of patterns, consisting of randomly distributed dotsand crossing strips, is introduced, where nodals and dots areincrementally identified from small neighborhoods. A remarkableadvantage of this combination is a substantially increased and moreuniformly distributed coordinates across each range frame.

2. Disambiguate between the two intersecting pattern strips fromrespective nodal and surrounding dots, in absence of any nodalconnectivity knowledge. Because strips in the image frame haveprojective consistency, the strips at a nodal are disambiguated from theorder of pixels around respective nodal, above and below epipolar line.

In cases where only one strip is visible on any side of the epipolar,identity is established from adjacent dots position.

Dot and strip patterns can originate from single radiation projector orfrom independent projection means having common optical axis. Dots andstrips may originate from pixels in a projector frame or fromilluminating printed transparencies.

In accordance to another method of present invention, a pattern ofrandomly distributed dots is projected onto a scene, where at least aportion of illuminated scene is recorded in a digital frame, where dotcoordinates are uniquely identified, and where frame coordinates arecalculated by triangulation thereof.

In yet another method of present invention, a scene is illuminated by acombination non-crossing strips and randomly distributed dots, projectedalong same optical axis, where dots and strips are uniquely identifiedfrom a digital image, and where frame coordinates are calculated bytriangulation thereof.

In accordance to embodiments of present invention there is provided anapparatus including:

projection means where at least one pattern is projected onto the sceneby said projection means;

at least one image sensor device configured to capture at least aportion of illuminated scene in an image frame; and

computing means configured to process image frame captured by the imagesensor to uniquely identify pattern features from illuminated pixels,where range coordinates are calculated thereof;

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of a scene digitizer system implementing afirst method of present invention having a projection device radiatingan object by crossing strip and dot patterns, fixed relative to an imagesensor, capturing reflected radiation, electrically coupled to acomputer having a network connection.

FIG. 2 is an exemplary depiction of strip and dot pattern in projector'sframe.

FIG. 3 is an exemplary depiction of strip and dot pattern captured atimage sensor's frame.

FIG. 4 is a schematic diagram of a scene digitizer system implementing asecond method of present invention, having a projection device radiatingan object by a random dot pattern, fixed relative to an image sensor,capturing reflected radiation, electrically coupled by a computer havinga network connection.

FIG. 5 is an exemplary depiction of random dot pattern in projector'sframe, showing exemplary epipolar regions subject to voting scheme.

FIG. 6 is an exemplary depiction of random dots captured at imagesensor's frame, showing an exemplary dot and neighborhood.

FIG. 7 is an exemplary depiction of random dots and non-crossing stripsin projector's frame.

FIG. 8 is an exemplary depiction of random dot and non-crossing strip inimage sensor's frame.

FIG. 9 is an exemplary depiction of random dots and crossing features.

DESCRIPTION OF PREFERRED EMBODIMENTS

FIG. 1 represents schematically a scene digitizer setup 100, whereprojector 102 emanates an optical radiation pattern from frame 116, ontothe scene, under computer 106 control. Projection pattern comprises twosets of strips mostly parallel, crossing each other, represented bysolid and dashed segments respectively, and a set of non-overlappingdots 117, having coordinates chosen at random.

At least a portion of the scene 110 is illuminated by the opticalradiation emanating from projector 102.

A digital image of at least a portion of reflected radiation is capturedby camera 104 and recorded at image frame 118 under computer 106control, which has connecting means to network 108. The portion ofreflected dots 122 are distinguishable groupings of pixels recorded atframe 118.

For the following explanation the term dot is substitute for dot'sposition coordinate.

FIG. 2 depicts strips in frame 116 exemplary represented byP_(A1)P_(An), and P_(B1)-P_(Bn), and exemplary nodals 206, 216.

FIG. 3 depicts strips in frame 118 exemplary represented byL_(A1)-L_(A2) and L_(B1)-L_(B2) which correspond to at least a subset ofP_(A1)-P_(An), and P_(B1)-P_(Bn), and exemplary nodals 306, 316.

When nodals in 118 are identified by carrying out the techniques asdescribed in co-pending application, it is possible to devise a stripconfiguration (periodicity, angularity) such that unambiguousidentification exists, i.e. no two nodals share same epipolar band.Practical application can require unrestricted configurations, forexample, to allow configuration changes dynamically.

In accordance to the first method of present invention, there isprovided a step of calculating coordinates of dots 122 in image framewith sub-pixel accuracy. This can be achieved, for example, bysegmenting pixels of each dot and computing corresponding centers ofgravity. Nodal identification is carried out according to method ofco-pending application, and includes determination of “colliding”nodals, i.e. nodals on same epipolar band. Further, colliding nodals aredisambiguated as described bellow.

To illustrate the method, nodals 206, 216 share epipolar Ep, and nodals306, 316 share epipolar Ec.

Dots in proximity of 206, 216 and 306 are searched and subsets 202, 212and 302 respectively, selected.

Subsets are further divided in two groups, with one group on each sideof epipolar line passing through respective nodal. For example, subsets202, 212 are divided by epipolar Ep; subset 302 is divided by epipolarEc. Match-test of a particular dot is performed over correspondinggroups. For example, a dot in 302 above epipolar Ec is match-testedagainst dots above epipolar line Ep in 202 and 212.

To carry out disambiguation, epipolar match-test is performed over atleast a fraction dots in each candidate set, counting the number ofmatches at each set.

When a predetermined majority of counts (votes) is collected in favor ofa single candidate, nodal identity is assigned to that candidate.

The minimum number of votes is subject to tradeoffs between processingspeed and “enough statistics” discriminating between nodal candidates.

The process is repeated until all colliding nodals are identified.

Disambiguation has low computational complexity because it consists offew simple operations and can be performed very fast since it lendsitself to parallel processing.

This way, dots in vicinity of all nodal points get identified. Theidentities of dots further away can be obtained in proximity ofpreviously identified dots, by choosing neighborhoods which includesubsets of identified and unidentified coordinates.

Because of random distribution, there is a high probability of obtainingunique matches amongst closest dots, in a small number of calculations.

Advantageously, the number of identifications increases with each step,as more test neighborhoods are created by expansion.

The search for nearest neighbors in image frame is facilitated by anumber of well-known techniques, one example being Kd-trees. Nearestneighbors in projector frame can be obtained at an initialization step,because relative positions do not change for a given patternconfiguration.

An alternative to expansion technique is searching for matchingneighborhoods along epipolar direction, where best match is dictated bya majority of successful match-test.

Another aspect addressed by this invention with respect to co-pendingapplication is improving on ascertaining individual strip identities ateach nodal, in absence of nodal-to-nodal connectivity graph.

To this end, nodal neighborhoods are analyzed to asses the presence ofdistinguishable strips on each side of local epipolar. If both stripsare detected on any side, their identities correspond directly to theorder in the pattern for respective nodal.

However, visibility may be such that image pixels contain insufficientstrip information to conclude on strip order, where only pixels of asingle strip, on any side of epipolar, are detected. To uniquelyidentify the singular strip, the configuration of neighborhood dots isemployed to pinpoint strip identity at that nodal.

At a nodal location, pattern strips divide each side of epipolar intothree regions, each encompassing a number of dots. Because dots positionwith respect to pattern strips is known, at a setup step, dots areassigned a supplemental attribute indicative of region they belong to.

To determine the identity of the singular strip, dots located on bothsides of the strip are searched and region data retrieved. Then thesigned distance between dots and strip pixels is calculated and itsidentity established form region data adjacency, i.e. identity isassigned to the strip delimiting the regions of mixed and un-mixed classof dots resides.

It will be appreciated that the embodiment described above is cited byway of example and that present method is not limited to what has beenparticularly shown and described. Rather, the scope of present inventionincludes combination and limit-cases of various features presentedhereinabove.

An example comprises a strip pattern which has one set ofnon-intersecting strips, and random dots, having a predetermined numberof small rectilinear segments crossing the strips at predeterminedlocations, where crossing features in image frame are set incorrespondence following nodal identification techniques describedabove, and expand dots identification thereof.

In accordance to a second method of present invention and in referenceto FIG. 4, camera-projector setup 400 is configured to illuminate object410 with pattern of randomly distributed dots 401 emanating fromprojection device 402, which has electrical coupling means to imagesensor 404 and computer 406. Radiation reflected off object 410 iscollected at image sensor 404, and stored at image frame 418, whereimage dots appear in the form of distinguishable pixels formations 414,characterized by coordinates of center of gravity at eachdistinguishable pixel formation.

Reference is now being made to FIG. 5 and FIG. 6

Pattern positions of image dots 600 can be carried out by searching dotsin small neighborhoods along epipolar lines, where a measure ofsimilarity is calculated at certain locations. Similarity measures canbe obtained in a number of ways, with varying degrees of computationalcomplexity, and carried out in frequency or spatial domain.

The method of present invention employs match-testing and votes countingto pinpoint identities in a small number of computations.

Because of discrete nature of pattern 500, epipolar searches isperformed around dots common to epipolar line 504, for example dots 502,504. In addition, in order to minimize errors, pattern 500 may beuncorrelated, to help robust discrimination of similarity measures.

A dot 602, from image set 600 is chosen at random, and a subset of dotsin predefined vicinity 601 collected and neighborhoods 501, 503 of matchcandidates 502, 504 respectively are searched.

Epipolar match-test calculations are performed over 501, 503, where foreach match a vote is counted. The neighborhood with most number of votesis chosen as winning candidates and respective dots identified thereof.

Speedup is achieved by limiting the calculations to correspondingsubsets on respective epipolar sides, as described in the first method.

For example, a dot in 601 located above epipolar 604 is probed againstdots in 501 and 503 above epipolar 504.

The number of dots in each neighborhood 501, 503, 601, is chosen toprovide sufficient statistics in deciding correct match by voting.Percentage of votes is linked to the ratio of image dots to pattern dotsand is chosen as probability of having all neighborhood dots matched.

At least a portion of winning neighborhood is identified from matchingprocess.

Further, identification by expansion is carried out, where an identifieddot is chosen such that a predefined vicinity includes a subset ofunidentified dots. Then the dot is match-tested against the chosen dot'sneighborhood in projector frame.

Neighborhood size in projector frame is chosen to contain at least anequal amount of dots as in image frame neighborhood, or a dynamicallyadjustable percentage.

Identification by expansion requires increasingly fewer operations andas such performs much faster because an increasingly smaller number ofmatch-tests are required to identify remaining dots.

The process has low computational complexity and because operations arelocalized in small regions, the process is suitable for parallelcalculations, where a large number of neighborhoods candidate searchesare supplied to independent computing units.

Other forms of computing similarity measures at distinct epipolarlocations exist and are considered to be in the scope of presentinvention.

Reference is now made to FIG. 9.

In an embodiment, the scene is illuminated by pattern 900 having asubset of randomly distributed dots interspersed with a predeterminednumber of distinguishable crossing features (crossings), whereidentification starts at and around crossings features and furtherexpands to remaining dots.

Crossing features can be positioned advantageously to minimize oreliminate candidate search process which speeds up identificationsignificantly and reduce identification errors.

Reference is now being made to FIG. 7 and FIG. 8.

In accordance to a third method of present invention, the projectionpattern comprise a set of randomly distributed dots 701 having a setrandom dots interspersed with of a set of non-crossing strips 700, wherethe non-crossing strips have an orientation mostly perpendicular toprojector-sensor plane, positioned such that two consecutive strips spana subset (band) of dots 701, where each dot is assigned bandinformation, indicative of bordering strip indices, e.g. L_(i), L_(i+1).

At least a portion of radiation reflected from object surface, whichcontains a subset of strips 700 and random dots 701, is captured at animage sensor.

At a first step, strips and dots are detected and located in image frame800 with sub-pixel accuracy.

At a second step, the dot set is searched and pattern positions locatedby way of the method described above.

At a third step, a predetermined number of pixels on strip segmentsS_(i), S_(j), are selected at random. For example a pixel on S_(i) ischosen and neighborhood 802 searched.

Dots 802 are looked up and “band” information retrieved. A tentativeidentity is assigned when dots on both sides of the strips belong toadjacent bands.

The procedure repeats at selected pixels and final identity of thesegment is chosen by majority vote, where at least a predeterminedpercent of locations returned same result.

A remarkable advantage of mixed patterns (dots and strips) technique isthe unique ability to obtain large numbers of frame measurements quicklyby localized sampling as well as profiling, and as such capture geometryfeatures and surface details, which advantageously complement eachother.

It will be appreciated that many modifications, variations andimprovements may occur to the skilled in the art. For example, a dotpattern and a singular strip is a particular case of disclosed method.This configuration can be suitable for a large number of scanningapplications, where successive profile samples are assembled in commonreference frames facilitated by the support of a large number of dotsamples.

In another configuration, a dot pattern and two crossing strips havingone intersection point, strips are quickly identified from the relativedirections in image frame, which results in an increased number ofscanning profiles.

In yet another configuration, a dot pattern and two parallel strips,close to each other, having no dots located in between the two strips,the two strips are identified from relative positions and/or fromrelative positions of dots.

The possible modification, variations and improvements following theprinciples of the invention, fall within the spirit of presentdisclosures and are covered by the present invention.

In an embodiment, projector-sensor system is configured to radiate anumber of projection patterns containing dots and strips, projected insuccession, where relative strip positions in two consecutive frames areincrementally shifted with respect to image sensor.

When a scene is illuminated by successive patterns a large number ofsampling profiles can be collected over a relatively small number ofconsecutive frames. The profiles can be subsequently assembled byalignment means, and scene (object) model obtained thereof.

Alignment is facilitated by the large number of samples in each frameand can be carried out by method well known in the art.

Systems configured for hand-held operations benefit substantially fromthe methods of present invention, enabling unconstrained and unattachedoperations, delivering detailed geometries of unprepared surfaces.

The present invention has significant advantages for mobile devices(e.g. tablets, smart phones) applications, because digitization can beachieved in a relatively small number of computations and by simple andlow-cost means, such as embedded camera and projector, which becomeincreasingly present in standard configurations.

Such devices can be configured such that frame to frame depth mapchanges may be indicative, for example, of certain commands instructinga device to execute certain actions.

Security applications benefit from present invention for depth-basedface recognition.

To achieve ambient light immunity, projected radiation is preferablyoutside visible spectrum, such as short-wave infrared in the range700-740 nm (nanometers), which is undetectable by human eye, butdetectable by an increasing number of common CMOS monochromatic sensorswith good photon efficiency in that regions of the spectrum.

Surface color can optionally be obtained in a number of modalities. Onemodality is radiometric decomposition, where projected patterns containknown amounts of radiation wavelengths, where radiation intensityreflected from the scene at a point is proportional with the amount ofcolor at that point. An example is color laser light, utilized in anumber of commercial projection engines, where the proportion of primarycolors can be controlled.

In an embodiment image sensor is a color camera, where each pixeldigitizes values of each color component. Determination of the exactcolor at each pixel is subject to a radiometric calibration step.

Objects characterized by weak geometry (featureless) may adverselyimpact align results because a unique matching solution may not exits,and as such incorrect 3-D models may result.

One way to address this issue is by including landmarks in the scene,which can be captured by the color sensor, and tracked at consecutiveframes, from which system's spatial positions (coordinates andorientation) are inferred and alignment performed thereof.

Another way to aid local alignment is by including an inertial sensor,capable to determine relative motion for short durations. Inertialsensors are increasingly commercially available, and are capable todeliver accurate spatial positions that can aid in aligninggeometrically challenged surfaces.

The presence of affixed artifacts in the scene is not necessary for thesystem to work, and are not part of present invention. Also, inertialsensors are not necessary for the system to perform and are not includedin this invention.

Inertial sensors and laser projectors may be more and more frequentlyfound in various mobile platform configurations, and as such, the methodof present invention are a good fit for spatial sensing applications onan increasing number of devices.

In an embodiment, surface color is obtained by configuring a colorcapable and a monochromatic image sensors having overlapping field ofviews, to record digital frames simultaneously or sequentially.

In this setup, the monochromatic sensor can be sensitive to a portion ofprojected radiation spectrum, which may or may not be visible radiation,obtaining frame coordinate thereof, whereas the color sensor can besensitive to a non-overlapping portion of the spectrum, which may or maynot be contained in projected radiation.

Color information can be assembled from a plurality of pixel intensitiesrecorded by the color sensor and possibly but not necessarily, fromintensities recorded by monochromatic sensor.

1. A method for obtaining three-dimensional measurements of physicalobjects comprising steps of: (a) projecting a plurality of radiationpatterns onto a scene, wherein said patterns comprise a plurality ofdistinct rectilinear strips, wherein said strips have substantiallyconstant intensity, wherein said strips of each said pattern are mostlyparallel, wherein said strips of each said pattern have predetermineddirections, wherein said strips of said patterns intersect other saidstrips at a plurality of points, wherein said plurality of patterns areinterspersed with distinguishable focal dots, wherein said dots arepositioned at random locations; (b) capturing at least one image ofreflected radiation from said scene, wherein said image comprises atleast a subset of reflected said strips, said intersection points, andsaid dots; (c) determining pattern positions of said some of the stripsand said some of the dots from said image locations of said intersectionpoints and said dots; (d) computing range measurements for said some ofthe strips and said some of the dots from said image locations; wherebysubstantially dense three-dimensional measurements are obtained, wherebysimple code-free patterns are employed, and whereby mobile scenes aredigitized.
 2. A method for digitization three-dimensional scenescomprising steps of: (a) projecting a plurality of distinguishable focaldots onto a scene, wherein said dots are positioned at random locations,wherein said dots do not overlap, wherein said dots have substantialdistribution density; wherein said dots may have interspersed crossingfeatures; (b) capturing an image of reflected radiation from said scene,wherein said image comprises at least a subset of said dots; (c)determining pattern positions of said some dots from image locations ofsaid some of the dots; (d) computing range measurements for said some ofthe dots from said image locations; whereby substantially densethree-dimensional measurements are obtained, and whereby mobile scenesare digitized.
 3. A digitization system comprising: (a) means forprojecting an optical radiation pattern onto a scene, wherein, saidpattern comprise a plurality distinguishable focal dots, wherein saiddots are positioned at random locations, wherein said dots do notoverlap, wherein said dots have substantial distribution density,wherein said dots may have interspersed distinguishable non-crossingstrips, and wherein location of said dots and said non-crossing strips,are under computer control. (b) an image sensor configured to captureoptical radiation reflected from said scene in at least one image,wherein said image comprises some of the said dots and some of the saidnon-crossing strips; (c) computing means configured to: i. determinepattern locations of said some of the dots and said some of thenon-crossing strips from said image locations; ii. calculate rangemeasurements for said some of the dots and said some of the non-crossingstrips from said image locations; (d) means for electrical coupling andmechanical controllability of said radiation pattern projection means,said image sensor and said computing means; (e) means for datacommunication; (f) means for graphic data display; whereby substantiallydense range frames are obtained, whereby range frames are obtained atsensor's frame rate, and whereby digitization of mobile scenes isobtained.